Aerial drone companion device and a method of operating an aerial drone companion device

ABSTRACT

A method of operating an aerial drone companion device includes detecting a first voice command spoken by a first user. The aerial drone companion device is autonomously oriented such that an image capture device faces the first user in response to detecting the first voice command. A second voice command spoken by the first user is detected while the image capture device faces the first user. The second voice command is transmitted from the aerial drone companion device to a computer located remotely from the aerial drone companion device. A task signal is received indicating a task to be performed. The task signal is generated by the computer based on the second voice command, and the task signal is transmitted by the computer and received by the aerial drone companion device. The method includes autonomously executing the task by the aerial drone companion device.

BACKGROUND

Exemplary embodiments of the present invention relate to an aerial dronecompanion device. More particularly, exemplary embodiments of thepresent invention relate to a method of operating an aerial dronecompanion device.

An automated home, which may be referred to as a “smart home,” mayinvolve the control and automation of lighting, heating, ventilation,air conditioning (HVAC), appliances, and security. For example, a smarthome may include switches and sensors connected to a central hubsometimes called a “gateway” from which the system is controlled with auser interface that is interacted either with a wall-mounted terminal,mobile phone software, tablet computer or a web interface.

A smart home may include one or more automated devices which respond tovoice commands. The automated devices may include an intelligentassistant which aids in the completion of household tasks. However,current intelligent assistants are generally static devices which arenot able to move about the home.

Some intelligent assistants may be mobile robots with the ability tomove about a residential or commercial space. However, such devices arelimited in their ability to move between different floors or differentelevations. For example, mobile robots may be limited in their abilityto travel from a ground level of a home to a second or third floorseparated by a staircase. Additionally, mobile robots may be limited intheir ability to cover relatively large distances while travelingoutdoors.

SUMMARY

Exemplary embodiments of the present invention provide a method ofoperating an aerial drone companion device includes detecting a firstvoice command spoken by a first user by at least one microphone disposedon the aerial drone companion device. The aerial drone companion deviceis autonomously oriented such that an image capture device disposed onthe aerial drone companion device faces the first user in response todetecting the first voice command. A second voice command spoken by thefirst user is detected by the at least one microphone while the imagecapture device faces the first user and while the first user looks atthe image capture device. The second voice command is transmitted fromthe aerial drone companion device to a computer located remotely fromthe aerial drone companion device. A task signal is received indicatinga task to be performed. The task signal is generated by the computerbased on the second voice command, and the task signal is transmitted bythe computer and received by the aerial drone companion device. Themethod of operating an aerial drone companion device includesautonomously executing the task by the aerial drone companion device.

According to an exemplary embodiment of the present invention, the firstvoice command may be a wakeup command that causes the aerial dronecompanion device to transition from a low-power operation mode to anormal operation mode.

According to an exemplary embodiment of the present invention, theaerial drone companion device may be stationed on ground while in thelow-power operation mode. The aerial drone companion device mayautonomously fly above the ground and follow the first user while in thenormal operation mode and while the user is moving. The aerial dronecompanion device may be stationed on the ground while in the normaloperation mode and while the user is not moving.

According to an exemplary embodiment of the present invention, thewakeup command may be predefined by the first user.

According to an exemplary embodiment of the present invention,autonomously orientating the aerial drone companion device may includeat least one of autonomously flying the aerial done companion devicetoward the first user and autonomously rotating the aerial dronecompanion device such that the image capture device faces the firstuser.

According to an exemplary embodiment of the present invention, thesecond voice command may indicate a destination at which the task is tobe executed. The aerial drone companion device may autonomously fly tothe destination to execute the task in response to receiving the tasksignal.

According to an exemplary embodiment of the present invention, thesecond voice command may indicate a time at which the task is to beexecuted. The aerial drone companion device may autonomously fly to thedestination at the indicated time to execute the task in response toreceiving the task signal.

According to an exemplary embodiment of the present invention, themethod of operating an aerial drone companion device may includegenerating a voice model of the first user using at least one of thefirst voice command, the second voice command, and additional voicecommands spoken by the first user. The method of operating an aerialdrone companion device may include storing the voice model of the firstuser in a first user record corresponding to the first user in a userdatabase. The method of operating an aerial drone companion device mayinclude generating a list of personalized preferences of the first userusing the second voice command and previous voice commands spoken by thefirst user. The method of operating an aerial drone companion device mayinclude storing the list of personalized preferences of the first userin the first user record.

According to an exemplary embodiment of the present invention, the tasksignal generated by the computer may be further based on the list ofpersonalized preferences of the first user.

According to an exemplary embodiment of the present invention, themethod of operating an aerial drone companion device may includecreating a voice model of a second user using at least one voice commandspoken by the second user. The method of operating an aerial dronecompanion device may include storing the voice model of the second userin a second user record corresponding to the second user in the userdatabase. The method of operating an aerial drone companion device mayinclude generating a list of personalized preferences of the second userusing the at least one voice command spoken by the second user. Themethod of operating an aerial drone companion device may include storingthe list of personalized preferences of the second user in the seconduser record. The method of operating an aerial drone companion devicemay include determining that the first and second user are present,using the at least one microphone, when the second voice command spokenby the first user is detected by the at least one microphone. The tasksignal generated by the computer may be further based on the list ofpersonalized preferences of the first user and the second user.

According to an exemplary embodiment of the present invention, thesecond voice command may direct the aerial drone companion device toplay music. The task signal may indicate a type of music to be playedbased on music preferences of the first user stored in the list ofpersonalized preference of the first user and on music preferences ofthe second user stored in the list of personalized preferences of thesecond user.

According to an exemplary embodiment of the present invention, themethod of operating an aerial drone companion device may includecapturing at least one image of the first user by the image capturedevice. The method of operating an aerial drone companion device mayinclude transmitting the at least one image of the first user from theaerial drone companion device to the computer located remotely from theaerial drone companion device. The method of operating an aerial dronecompanion device may include storing the at least one image of the firstuser in a first user record corresponding to the first user in a userdatabase. The method of operating an aerial drone companion device mayinclude generating a list of personalized preferences of the first userusing the second voice command and previous voice commands spoken by thefirst user. The method of operating an aerial drone companion device mayinclude storing the list of personalized preferences of the first userin the first user record.

According to an exemplary embodiment of the present invention, the tasksignal generated by the computer may be further based on the list ofpersonalized preferences of the first user.

According to an exemplary embodiment of the present invention, themethod of operating an aerial drone companion device may includecapturing at least one image of the first user by the image capturedevice and determining a current cognitive state of the first user usingthe at least one image. The task signal generated by the computer may befurther based on the current cognitive state of the first user.

According to an exemplary embodiment of the present invention, the taskmay include communicating with an appliance located remotely from theaerial drone companion device to execute a function of the appliance.

According to an exemplary embodiment of the present invention, the taskincludes placing an order for goods or services by relaying the orderfor goods or services from the aerial drone companion device to a selleror provider of goods or services located remotely from the aerial dronecompanion device.

According to an exemplary embodiment of the present invention, themethod of operating an aerial drone companion device may includefiltering noise generated by a propeller of the aerial drone companiondevice to identify the first and second voice commands spoken by thefirst user.

Exemplary embodiments of the present invention provide a method ofoperating an aerial drone companion device includes capturing at leastone registration image of a user by an image capture device disposed onthe aerial drone companion device and registering the user as a targetuser of the aerial drone companion device using the at least oneregistration image. The aerial drone companion device may beautonomously flown from a charging station to the target user. Theaerial drone companion device locates the target user by capturingimages using the image capture device while flying, and by comparingpeople in the captured images to the at least one registration imageuntil the target user is identified. The aerial drone companion devicecycles between autonomously docking in the charging station andautonomously flying to the target user at a predetermined interval. Themethod of operating an aerial drone companion device includesdetermining a current cognitive state of the target user by capturing atleast one image of the target user using the image capture device andcomparing the at least one image to a plurality of predefined images inan image database. A task is autonomously executed based on thecognitive state of the target user without receiving input from theuser.

According to an exemplary embodiment of the present invention, thecurrent cognitive state of the user may indicate that the target userrequires emergency assistance, and the task may include the aerial dronecompanion device contacting a third party.

Exemplary embodiments of the present invention provide an aerial dronecompanion device includes a processor, at least one microphone thatdetects a first voice command and a second voice command spoken by auser under control of the processor and an image capture device. Theaerial drone companion device includes a plurality of propellers thatautonomously orientate the aerial drone companion device, under controlof the processor, such that the image capture device faces the user inresponse to the at least one microphone detecting the first voicecommand. The second voice command is detected by the at least onemicrophone while the image capture device faces the first user and whilethe first user looks at the image capture device. The aerial dronecompanion device includes a network adapter that transmits the secondvoice command from the aerial drone companion device to a computerlocated remotely from the aerial drone companion device under control ofthe processor, and that receives a task signal from the computerindicating a task to be performed. The task signal is generated by thecomputer based on the second voice command. The aerial drone companiondevice autonomously executes the task.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features of the present invention will become moreapparent by describing in detail exemplary embodiments thereof, withreference to the accompanying drawings, in which:

FIG. 1 illustrates an aerial drone companion device according to anexemplary embodiment of the present invention.

FIG. 2 illustrates a method of operating an aerial drone companiondevice according to an exemplary embodiment of the present invention.

FIG. 3 illustrates a method of operating an aerial drone companiondevice according to an exemplary embodiment of the present invention.

FIG. 4 illustrates a method of operating an aerial drone companiondevice according to an exemplary embodiment of the present invention.

FIG. 5 illustrates a method of voice recognition by an aerial dronecompanion device according to an exemplary embodiment of the presentinvention.

FIG. 6 illustrates a method of facial recognition by an aerial dronecompanion device according to an exemplary embodiment of the presentinvention.

FIG. 7 illustrates a method of cognitive state identification by anaerial drone companion device according to an exemplary embodiment ofthe present invention.

FIG. 8 illustrates a method of storing and accessing user preferences byan aerial drone companion device according to an exemplary embodiment ofthe present invention.

FIG. 9 illustrates a method of operating an aerial drone companiondevice according to an exemplary embodiment of the present invention.

FIG. 10 illustrates an example of a computer system capable ofimplementing the methods according to exemplary embodiments of thepresent invention.

FIG. 11 depicts a cloud computing environment according to an embodimentof the present invention.

FIG. 12 depicts abstraction model layers according to an embodiment ofthe present invention.

DETAILED DESCRIPTION

It will be understood that the terms “first,” “second,” “third,” etc.are used herein to distinguish one element from another, and theelements are not limited by these terms. Thus, a “first” element in anexemplary embodiment may be described as a “second” element in anotherexemplary embodiment.

Exemplary embodiments of the present invention will be described morefully hereinafter with reference to the accompanying drawings. Likereference numerals may refer to like elements throughout thespecification and drawings.

A method and system according to an exemplary embodiment of the presentinvention may include a flying drone with voice input and output. Inresponse to detecting a spoken keyword, the drone may wake up and orienttoward the speaker of the keyword (e.g., the “user” of the drone). Basedon receiving further words from the user, while the user's gaze isdirected toward the drone, the drone may transmit a command to a cloudcomputing platform. Based on a signal returned from the cloud computingplatform, the drone both replies to the user and also flies to alocation in space (e.g. to play music 6 feet away on the porch of ahome).

A Companion Flying Drone (CFD) may include a wireless speaker and voicecommand module. The CFD may be capable of voice interaction, musicplayback, making to-do lists, setting alarms, streaming podcasts,playing audiobooks, and providing weather, traffic and other real timeinformation. The CFD may control several smart devices, including smartdevices in the home. The terms “companion flying drone” and “aerialdrone companion device” may be used interchangeably herein.

FIG. 1 illustrates an aerial drone companion device according to anexemplary embodiment of the present invention.

Referring to FIG. 1, according to an exemplary embodiment of the presentinvention, an aerial drone companion device 100 includes a processor121, at least one microphone 102 that detects a first voice command anda second voice command spoken by a user (see, e.g., user 301illustrated, for example, in FIG. 3) under control of the processor 121and an image capture device 103. The aerial drone companion device 100includes a plurality of propellers 101 that autonomously orientate theaerial drone companion device 100, under control of the processor 121,such that the image capture device 103 faces the user in response to theat least one microphone 102 detecting the first voice command. Thesecond voice command is detected by the at least one microphone 102while the image capture device 103 faces the first user and while thefirst user looks at the image capture device 103. The aerial dronecompanion device 100 includes a network adapter 105 that transmits thesecond voice command from the aerial drone companion device 100 to acomputer (see, e.g., on-board computer 120, or remote computer 320illustrated, for example, in FIG. 3) located remotely from the aerialdrone companion device 100 under control of the processor 121, and thatreceives a task signal from the computer indicating a task to beperformed. The task signal is generated by the computer based on thesecond voice command. The aerial drone companion device 100 autonomouslyexecutes the task.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may include a battery, a GPS module107, a Bluetooth connection module 106, at least one speaker 110, alight array 111 and an object manipulation arm 104. The GPS module 107may be used to identify a location of the aerial drone companion device100. The Bluetooth connection module 106 may provide Bluetoothcommunication between the aerial drone companion device 100 and otherdevices, such as the user's smart device or an appliance. The speaker110 may be used to play music or audio books. The speaker 110 may beused for the aerial drone companion device 100 to elicit voice responsesto the user. The object manipulation arm 104 may be used to operate anappliance.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may include a voice recognition module108. The voice recognition module may include a voice command interface130 and a control circuit 134 including a microphone 131, a levelconversion circuit 132 and a voice processing circuit 133. The voicerecognition module 108 is illustrated in FIG. 1, for example, as beingon-board the aerial drone companion device 100. However, the voicerecognition module 108 may also be stored in the cloud, and may beaccessed, for example, by the network adapter 105. The voice recognitionmodule 108 is described in more detail below.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may include an on-board computer 120.The on-board computer 120 may include an interaction information hub124, a processor 121, a memory 122 and a user database 123. The on-boardcomputer 120 is illustrated in FIG. 1, for example, as being on-boardthe aerial drone companion device 100. However, the on-board computer120 may be omitted, and the aerial drone companion device 100 maycommunicate with the remote computer 320. The on-board computer 120 maybe substantially the same as the remote computer 320 except for beingdisposed on the aerial drone companion device 100. The remote computer320 is described below in more detail.

According to an exemplary embodiment of the present invention, the imagecapture device 103 may include a built-in microphone 109. The built-inmicrophone 109 may be disposed on the image capture device 103 and maycapture one or more voice commands from one or more users according toexemplary embodiments of the present invention. The image capture device103 may also receive one or more voice commands from the microphone 102.Voice commands may be transmitted from the built-in microphone 109and/or the microphone 102 to the on-board computer 120 and/or to theremote computer 320.

According to an exemplary embodiment of the present invention, the lightarray 111 may include a plurality of lights. For example, the lightarray 111 may include red, yellow and green lights. The lights may beused to provide a signal from the aerial drone companion device 100 tothe user. For example, a green light may indicate that the aerial dronecompanion device 100 has a relatively full battery, while a red lightmay indicate the battery is almost exhausted. Additionally, the lightsmay be used to communicate whether a voice command has been receivedand/or interpreted correctly to a reasonable level of confidence, asdiscussed below in more detail.

According to an exemplary embodiment of the present invention, the voicecontrol circuit 134 of the voice recognition module 108 may include themicrophone 131, the level conversion circuit 132 and the voiceprocessing circuit 133. The voice control circuit 134 may include avoice operated switch connected between the microphone 131 and the levelconversion circuit 132. The microphone 131 may pick up voice commands,the voice operated switch may receive the voice commands from themicrophone 131, and output a high voltage signal when a volume of thevoice commands is greater than or equal to a predetermined volumethreshold or is within a predetermined volume range. The levelconversion circuit 132 may convert the high voltage signal into a lowvoltage signal for turning on the aerial drone companion device 100. Thevoice command interface 130 need not be activated until a message (e.g.,an e-mail, a text message, or a voice mail) has been received by theaerial drone companion device 100. The arrival of a message may be usedto trigger the voice command interface 130 by activating one or morespeech recognition routines in a predetermined time period correspondingto the one or more speech recognition routines. The voice commandinterface 130 may signal the aerial drone companion device 100 to turnoff, and/or return to a charging station and/or enter a low power/sleepmode when the predetermined time period expires or the user has nofurther commands.

According to an exemplary embodiment of the present invention, ashort-range range wireless data and voice communication link, forexample using Bluetooth technology, may be established between a remotecontrol device and the aerial drone companion device 100. To activatevoice control circuitry in the aerial drone companion device 100, a usermay supply an input to the remote control device. In response to theuser input, wireless link circuitry within the remote control device maysend a control signal, for example an AT command, to the aerial dronecompanion device 100. The aerial drone companion device 100 may receivethe control signal and transmits a control signal indicating the statusof the aerial drone companion device 100. Once the voice controlcircuitry is activated, the user may provide a voice command to theremote control device. The remote control device may transmit the voicesignal to the aerial drone companion device 100 over the wirelesscommunication link. The aerial drone companion device 100 may processthe voice signal in the voice control circuitry and generateinstructions in response to the voice signal. The voice controlcircuitry (e.g., the voice command interface 130 of the voicerecognition module 108) may be located within the aerial drone companiondevice 100 and not within the remote control device, which may simplifythe remote control device and may reduce the amount of power dissipatedin the remote control device.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may use the multimodalities of socialintelligent embodied cognitive agents' implementation to personalize thecompanion tasks by sensing and responding to users, objects, andemotions all by applying deep neural net and visual analyticstechniques, and by leveraging cloud computing capabilities.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may use or reuse planning andreasoning, including scripting, rule-based, and learned (e.g., viagenetic programming and reinforcement) while facilitating multimodalinteractions with the user. The aerial drone companion device 100 may beadapted to control multiple controlled drones in coordination so thateach coordinated drone is made to perform the same or a differentfunction in response to a single voice command. The aerial dronecompanion device 100 may learn user habits, as discussed in more detailbelow with reference to FIG. 8.

FIG. 2 illustrates a method of operating an aerial drone companiondevice according to an exemplary embodiment of the present invention.FIG. 3 illustrates a method of operating an aerial drone companiondevice according to an exemplary embodiment of the present invention.

Referring to FIGS. 2 and 3, according to an exemplary embodiment of thepresent invention, a method of operating an aerial drone companiondevice includes detecting a first voice command spoken by a first userby at least one microphone disposed on the aerial drone companion device201. The aerial drone companion device is autonomously oriented suchthat an image capture device disposed on the aerial drone companiondevice faces the first user in response to detecting the first voicecommand 202. A second voice command spoken by the first user is detectedby the at least one microphone while the image capture device faces thefirst user and while the first user looks at the image capture device203. The second voice command is transmitted from the aerial dronecompanion device to a computer located remotely from the aerial dronecompanion device 204. A task signal is received indicating a task to beperformed 205. The task signal is generated by the computer (e.g., theremote computer 320) based on the second voice command, and the tasksignal is transmitted by the computer and received by the aerial dronecompanion device 100. The method of operating an aerial drone companiondevice 100 includes autonomously executing the task by the aerial dronecompanion device.

According to an exemplary embodiment of the present invention, speechrecognition and/or natural language processing may be employed ingenerating the task signal, as discussed below in more detail.

According to an exemplary embodiment of the present invention, the firstand second voice commands may be uttered by the user 301. The aerialdrone companion device 100 may receive the first command and may receivea first task signal from the remote computer 320 to leave a chargingstation 302, fly to the general vicinity of the user 301 and orient theimage capture device 103 toward the user's face to capture the user'sgaze. Alternatively, the first command may be parsed and the first tasksignal may be received from the on-board computer 120. Thus, the aerialdrone companion device 100 may fly toward the user 301 and may orientthe image capture device 103 toward the user's face.

According to an exemplary embodiment of the present invention, thecomputer (e.g, the remote computer 320) may receive input (e.g., in theform of voice, gesture, or an explicit command) and may apply one ormore analytics modules to recognize the type of task to be performed,and finally generate the task list. Based on the received input tasksignal form the one or more modules may utilize one or more analyticalgorithms and models, which may include voice recognition, gesturedetection (e.g., using deep learning or neural network), and/or visualanalytics. The models are trained or built using historical andlongitudinal task signals stored on one or more databases, as describedherein.

According to an exemplary embodiment of the present invention,determining whether the user's gaze is directed toward the aerial dronecompanion device 100 may help the aerial drone companion device'sadvanced imaging system increase the its confidence level in theidentification of the speaker of the command or query, and may help theaerial drone companion device 100 assess a possible additional cognitiveaspect of the command (e.g., based on facial expression). Even thoughthe user will typically use an initial trigger word like, “Drone, fly tothe head of the bed to wake me up in the morning,” the confidence levelin the command being directed to the drone will increase based on user'sgaze. For example, if the aerial drone companion device 100 is able todetect and register the user's gaze (e.g., for a predetermined amount oftime) it may increase a confidence level that the aerial drone companiondevice 100 is, in fact, intended to receive a command uttered by theuser.

According to an exemplary embodiment of the present invention the aerialdrone companion device 100 might not require a user's gaze to be focusedon the aerial drone companion device 100. For example, the user may befocused on a task requiring their hands (e.g., fixing something) and maynot want to look away, or the aerial drone companion device 100 may haveproactively initiated an interaction with a child or elderly person whomay not be so focused on the aerial drone companion device 100.Additionally, the user or individual interacting with the drone may havea visual or mobility impairment that makes gazing at the aerial dronecompanion device 100 difficult or impossible.

According to an exemplary embodiment of the present invention, the firstvoice command may be a wakeup command that causes the aerial dronecompanion device 100 to transition from a low-power operation mode to anormal operation mode. The wakeup command may be predefined by the firstuser (e.g., before the aerial drone companion device enters a low powermode or docks with the charging station—see, e.g. FIG. 8).

The low-power operation mode may be a form of a sleep mode in which thedrone uses a minimal or reduced amount of power, but is still capable ofreceiving or generating task signals. For example, the aerial dronecompanion device 100 may be landed/not flying when in low-power mode, ormay still be flying but have certain features (e.g., WiFi, Bluetooth,cell service, camera) turned off to save power. Normal operation modemay have substantially all hardware turned on and ready to acceptcommands and communicate with outside devices (e.g. an appliance) and/orthe remote computer 320. However, exemplary embodiments of the presentinvention are not limited thereto and the aerial drone companion device100 is not necessarily in low power operation mode when it is stationaryor not flying. That is, the aerial drone companion device 100 may bedocked with the charging station, connected to external power, orresting on the ground or on another surface (e.g., not flying), whilestill in normal operation mode (e.g., full or near-full power mode).Further, the aerial drone companion device 100 may be waiting for auser's command even when not in low-power operation mode (e.g., theaerial drone companion device 100 may be waiting for a user's commandwhile it is in normal operation mode). For example, the aerial dronecompanion device 100 may be docked to a power source and be in normalmode and waiting for a user's command.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may be docked with the chargingstation 302. The charging station may provide ground power to the aerialdrone companion device 100 while the aerial drone companion device 100is charging. The charging station 302 may include a wirelesscommunication antenna, may receive a signal (e.g., from the user) andmay communicate the signal to the aerial drone companion device 100without the use of the network adapter 105 aerial drone companion device100. Thus, the aerial drone companion device 100 may use little or nobattery power while connected with the charging station 302.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may be stationed on ground while inthe low-power operation mode. The aerial drone companion device 100 mayautonomously fly above the ground and follow the first user while in thenormal operation mode and while the user is moving. The aerial dronecompanion device 100 may be stationed on the ground while in the normaloperation mode and while the user is not moving. That is, the aerialdrone companion device 100 may follow the user. For example, the aerialdrone companion device 100 may use voice or facial recognition toidentify a primary user of the aerial drone companion device 100 andonce the primary user is identified, the aerial drone companion device100 may hover a predetermined distance away from the user asobstructions allow. In the event the primary user is lost, the aerialdrone companion device 100 may identify the location of the user via theuser's Smartphone or other smart device. Bluetooth, WiFi, cell towernetworks or other short range radio frequency communication may be usedto identify the location of a user's Smartphone or other smart device(e.g., Smart Watch). If a secondary user has most recently activated theaerial drone companion device 100 then the aerial drone companion device100 may follow that secondary user until the primary user issues acommand to the aerial drone companion device 100.

According to an exemplary embodiment of the present invention, the taskmay include communicating with an appliance (e.g., the appliance 103)located remotely from the aerial drone companion device to execute afunction of the appliance. The appliance may be one of a light and amedia device. The task may include turning the light on or off when theappliance is the light. The task may include executing a media functionof the media device when the appliance is the media device.

The aerial drone companion device 100 may receive a task command to turnon the appliance 103, which may be alight including an activation switch304. The aerial drone companion device 100 may reorient itself with theobject manipulation arm 104 temporarily coupled to or in contact withthe activation switch 304. The activation switch 304 may be moved (e.g.,in an upward direction) and the appliance may be turned on. Theappliance 303 may include a wireless or Bluetooth connection antenna 315and may communicate wirelessly with the aerial drone companion device100.

The aerial drone companion device may include the network adapter 105and/or the Bluetooth connection module 106. The aerial drone companiondevice 100 may connect to and communicate with a cell towercommunication network, may include WiFi connectivity capability or anyother short range radio communication capability, as desired.

The remote computer 320 may include a wireless connection antenna 305,an interaction information hub 324, a processor 321, a memory 322 and auser database 323. The remote computer 320, the interaction informationhub 324, the processor 321, the memory 322 and the user database 323 mayperform substantially the same functions as the on-board computer 120,the interaction information hub 124, the processor 121, the memory 122and the user database 123 described above with reference to FIG. 1. Theremote computer 320 will be described in more detail below withreference to FIGS. 5-8.

The remote computer 320 is illustrated as accessed through the cloud306. However, exemplary embodiments of the present invention are notlimited thereto. For example, the remote computer 320 might not beaccessed through the cloud.

According to an exemplary embodiment of the present invention, the tasksignal may be generated by the remote computer 320 by performing naturallanguage processing (NLP) to parse a received voice command (e.g., thesecond voice command). For example NLP may include parsing the voicecommand into words, searching a database for keywords, and determiningwhat the user is asking the drone to do based on the keywords. Forexample, a first user (User A—see, e.g., FIG. 8) may prefer a certaintype of music be played at a certain type of day. The user may beidentified by voice and/or facial recognition, as described according toexemplary embodiments of the present invention herein. The NLP processperformed by the remote computer 320 may identify the term “music” andmay identify the time of day. The combination of identifying User A, theterm “music” and a time of day may result in a command to play a certaintype of music. For example, as discussed in more detail below withreference to FIG. 8, if the time is 7:30 PM and User A is identified,upbeat music might be played.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may generate a task through learningmodels embodied in the drone's on-board computer 120. One or moreapplications may run on the aerial drone companion device 100 that usethe in-drone deployed learning models. The aerial drone companion device100 may use the remote computer 320 for advanced processing, advancedanalytics, and/or data storage. The in-drone learning models andalgorithms may be updated by fetching from the remote computer 320.

NLP may include sentence segmentation, speech tagging (e.g., part ofspeech tagging) and language parsing. Deep analytics may be employed toextract specific information from multiple score data sets, such as asegmented and tagged sentence. A database of keywords may be used toidentify keywords understood by the remote computer 320. Named entityextraction may be employed. Name entity extraction may be used for datamining. A named entity or definition (e.g., a word or phrase)identifying an item of interest may be searched in a tagged and parsedsentence. NLP may include automated summarization in which identifiedkeywords are combined to extract meaning from a tagged and parsedsentence. For example, the terms/phrases “music” “deck” “in 5 minutes”“hover at 5 feet above” may be identified and understood in a particularsentence spoken by a user and may lead the remote computer 320 totransmit a task signal of “travel to User A's deck in 5 minutes, howeverat 5 feet above the deck and play upbeat music.” The “upbeat” task maybe identified from User A's preferences, as discussed below in moredetail with reference to FIG. 8.

According to an exemplary embodiment of the present invention,autonomously orientating the aerial drone companion device 100 mayinclude at least one of autonomously flying the aerial done companiondevice 100 toward the first user and autonomously rotating the aerialdrone companion device 100 such that the image capture device 103 facesthe first user.

According to an exemplary embodiment of the present invention, thesecond voice command may indicate a destination at which the task is tobe executed. The aerial drone companion device 100 may autonomously flyto the destination to execute the task in response to receiving the tasksignal.

According to an exemplary embodiment of the present invention, thesecond voice command may indicate a time at which the task is to beexecuted. The aerial drone companion device 100 may autonomously fly tothe destination at the indicated time to execute the task in response toreceiving the task signal.

FIG. 4 illustrates a method of operating an aerial drone companiondevice according to an exemplary embodiment of the present invention.

Referring to FIG. 4, according to an exemplary embodiment of the presentinvention, a method of operating an aerial drone companion deviceincludes capturing at least one registration image of a user by an imagecapture device disposed on the aerial drone companion device andregistering the user as a target user of the aerial drone companiondevice using the at least one registration image 401. The aerial dronecompanion device is autonomously flown from a charging station to thetarget user 402. The aerial drone companion device locates the targetuser by capturing images using the image capture device while flying,and by comparing people in the captured images to the at least oneregistration image until the target user is identified 403. The aerialdrone companion device cycles between autonomously docking in thecharging station and autonomously flying to the target user at apredetermined interval 404. The method of operating an aerial dronecompanion device includes determining a current cognitive state of thetarget user by capturing at least one image of the target user using theimage capture device and comparing the at least one image to a pluralityof predefined images in an image database 405. A task is autonomouslyexecuted based on the cognitive state of the target user withoutreceiving input from the user 406.

According to an exemplary embodiment of the present invention, acaptured user image may be compared with stored user images, asdiscussed in more detail below with reference to FIG. 6.

According to an exemplary embodiment, the aerial drone companion device100 may be set to check on a user (e.g., an elderly user or a child) atset intervals (e.g., every 30 minutes). For example, as discussed belowin more detail, in addition to facial expression analysis, the aerialdrone companion device 100 may detect a posture or body position of auser or another individual (e.g., to identify if someone is on theground and potentially injured).

According to an exemplary embodiment, the method of operating the aerialdrone companion device 100 may include filtering noise generated by thepropellers 101 of the aerial drone companion device 100 to identify thefirst and second voice commands spoken by the first user. For example, apitch or angle of the propellers 101 may be adjusted to minimize noisegenerated by the propellers 101. Alternatively, the aerial dronecompanion device 100 may land on a nearby object and reduce or stop arotation of the propellers 101 to better hear a user's command.Additionally, a speed of the engines rotating the propellers 101 may bereduced to reduce noise generated by the propellers 101.

Reducing or filtering noise generated by the propellers 101 of theaerial drone companion device 100 may include active noise control(ANC), which may also be referred to as noise cancellation or activenoise reduction (ANR). ANC may reduce unwanted sound by the addition ofa second sound specifically designed to cancel the first. Alternatively,reducing or filtering noise generated by the propellers 101 of theaerial drone companion device 100 may include Adaptive broadbandfiltration, Adaptive inverse filtration, Frequency Compensation Impulsefiltration, Dynamic processing, and/or Stereo processing. Speakeridentification, speech enhancement and audio restoration algorithms mayalso be employed, as described herein.

FIG. 5 illustrates a method of voice recognition by an aerial dronecompanion device according to an exemplary embodiment of the presentinvention.

Referring to FIG. 5, according to an exemplary embodiment of the presentinvention, the remote computer 320 and/or the on-board computer 120 ofthe aerial drone companion device 100 may perform voice recognition ofone or more users. The method of operating an aerial drone companiondevice 100 may include generating a voice model of the first user (see,e.g., voice identification profile for User 1, User 2 and User 3illustrated in FIG. 5) using at least one of the first voice command,the second voice command, and additional voice commands spoken by thefirst user. The method of operating an aerial drone companion device 100may include storing the voice model of the first user in a first userrecord corresponding to the first user in a user database 323. Themethod of operating an aerial drone companion device may includegenerating and maintaining a list of personalized preferences of thefirst user (see, e.g., FIG. 8) using the second voice command andprevious voice commands spoken by the first user. The method ofoperating an aerial drone companion device may include storing the listof personalized preferences of the first user in the first user record(see, e.g., FIG. 8). The generation, tabulation and storage of userpreferences is discussed below in more detail with reference to FIG. 8.

According to an exemplary embodiment of the present invention, a speechrecognition algorithm may be employed for identifying one or more users.For example, speech recognition may employ a Hidden Markov Model,dynamic time warping (DTP)-based speech recognition, neural networks,deep feedforward and/or recurrent neural networks.

According to an exemplary embodiment of the present invention, themethod of operating an aerial drone companion device 100 may includecreating a voice model of a second user using at least one voice commandspoken by the second user. The method of operating an aerial dronecompanion device may include storing the voice model of the second userin a second user record corresponding to the second user in the userdatabase 323. The method of operating an aerial drone companion device100 may include generating a list of personalized preferences (see,e.g., FIG. 8) of the second user using the at least one voice commandspoken by the second user. The method of operating an aerial dronecompanion device 100 may include storing the list of personalizedpreferences (see, e.g., FIG. 8) of the second user in the second userrecord. The method of operating an aerial drone companion device 100 mayinclude determining that the first and second user are present, usingthe at least one microphone (e.g., microphone 102), when the secondvoice command spoken by the first user is detected by the at least onemicrophone. The task signal generated by the computer (e.g., theon-board computer 120 and/or the remote computer 320) may be furtherbased on the list of personalized preferences of the first user and thesecond user. The generation, tabulation and storage of user preferencesis discussed below in more detail with reference to FIG. 8.

According to an exemplary embodiment of the present invention, a voicerecognition module 501 may identify one or more users by voicerecognition. A voice identification profile may be stored for each of aplurality of users. A captured voice recording of a user may be used toidentify the user. The on-board computer 120 and/or the remote computer320 may compare the captured voice recording and a list of stored voiceidentification profiles stored in the user database 323. A voiceidentification threshold may be set at any desired level. The voiceidentification threshold may be a minimum similarity level at which acaptured voice recording is considered sufficiently the same as a storedvoice identification profile to result in a match and identification ofa stored user. For example, the voice identification threshold may beset at 90%. The on-board computer 120 and/or the remote computer 320 maycompare the captured voice recording and a list of stored voiceidentification profiles stored in the user database 323. If the capturedrecording profile and the stored voice identification profile match at alevel of 90% or higher, then the corresponding user is identified. Thesimilarly between a plurality of data points on the line graphsillustrated in FIG. 5 may be compared between the stored profiles andthe captured profile and on a scale of 0-100. For example, 90/100 datapoints matching would result in a voice match profile score of 90%. Ifthe voice match profile score is at least 90% when the identificationthreshold is set to 90% then a match is made of the corresponding user.In the example illustrated in FIG. 5, the threshold is set at 90% andthe profile match score for User 2 is 99%, and thus User 2 is identifiedas having uttered the captured voice recording. However, exemplaryembodiments of the present invention are not limited thereto, and voiceidentification analysis may be performed as desired.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may determine an approximate locationof a user's spoken voice (e.g., to better reorient itself to face theuser and to detect the user's gaze, as discussed herein in more detail).Source direction may be detected using the time difference of arrival(TDOA) method, which may employ one or more pressure microphones and/orparticle velocity probes. For example, the microphones described herein(e.g., 102, 109 and 131) may be pressure microphones).

With a sensor array (e.g., a microphone array including more than onemicrophone) consisting of at least two probes it is possible to obtainthe source direction using the cross-correlation function between eachprobes' signal. The cross-correlation function between two microphonesis defined as

${{R_{{x\; 1},{x\; 2}}(\tau)} = {\sum\limits_{n = {- \infty}}^{\infty}\; {{x_{1}(n)}{x_{2}( {n + \tau} )}}}},$

which defines the level of correlation between the outputs of twosensors x_1 and x_2. In general, a higher level of correlation meansthat the argument \tau is relatively close to the actualtime-difference-of-arrival. For two sensors next to each other the TDOAis given by

$\tau_{true} = \frac{d_{spacing}}{c}$

where c is the speed of sound in the medium surrounding the sensors andthe source.

An example of TDOA is the interaural time difference. The interauraltime difference is the difference in arrival time of a sound between twoears. The interaural time difference is given by

${\Delta \; t} = \frac{x\; \sin \; \theta}{c}$

where \Delta t is the time difference in seconds, x is the distancebetween the two sensors (ears) in meters, \theta is the angle betweenthe baseline of the sensors (ears) and the incident sound, in degrees.

FIG. 6 illustrates a method of facial recognition by an aerial dronecompanion device according to an exemplary embodiment of the presentinvention.

Referring to FIG. 6, according to an exemplary embodiment of the presentinvention, a facial recognition module 601 may identify one or moreusers by facial images. A facial image may be stored for each of aplurality of users. A captured facial image (e.g., captured by the imagecapture device 103) may be used to identify the user. The on-boardcomputer 120 and/or the remote computer 320 may compare the capturedfacial image and a list of stored facial images stored in the userdatabase 323. A facial identification threshold may be set at anydesired level. The facial identification threshold may be a minimumsimilarity level at which a captured facial image is consideredsufficiently the same as a stored facial to result in a match andidentification of a stored user. For example, the facial identificationthreshold may be set at 90%. The on-board computer 120 and/or the remotecomputer 320 may compare the captured facial image and a list of storedfacial images stored in the user database 323. If the captured facialimage and the stored facial image match at a level of 90% or higher,then the corresponding user is identified. The similarly between aplurality of data points on the captured facial image may be comparedbetween the stored facial image and the captured facial image and on ascale of 0-100. For example, 90/100 data points matching would result infacial image identification match score of 90%. If the facial imageidentification match score is at least 90% when the identificationthreshold is set to 90% then a match is made of the corresponding user.In the example illustrated in FIG. 6, the threshold is set at 90% andthe profile match score for User 3 is 98%, and thus User 3 isidentified. However, exemplary embodiments of the present invention arenot limited thereto, and facial recognition analysis may be performed asdesired.

FIG. 7 illustrates a method of cognitive state identification by anaerial drone companion device according to an exemplary embodiment ofthe present invention.

Referring to FIG. 7, according to an exemplary embodiment of the presentinvention, the method of operating an aerial drone companion device 100may include capturing at least one image of the first user by the imagecapture device 103 and determining a current cognitive state of thefirst user using the at least one image. The task signal generated bythe computer (e.g., 120 or 320) may be further based on the currentcognitive state of the first user.

According to an exemplary embodiment of the present invention, acognitive state identification module 701 may identify one or morecognitive states of a user (e.g., happy, sad or frustrated). A pluralityof facial images corresponding with different cognitive states may bestored in the user database 323. A captured facial image (e.g., capturedby the image capture device 103) may be used to identify the cognitivestate of a user. The on-board computer 120 and/or the remote computer320 may compare the captured facial image and a list of stored facialimages stored in the user database 323. A cognitive state identificationthreshold may be set at any desired level. The cognitive stateidentification threshold may be a minimum similarity level at which acaptured facial image is considered sufficiently the same as a storedfacial to result in a match and identification of a cognitive state. Forexample, the cognitive state identification threshold may be set at 90%.The on-board computer 120 and/or the remote computer 320 may compare thecaptured facial image and a list of stored facial images stored in theuser database 323. If the captured facial image and the stored facialimage match at a level of 90% or higher, then the correspondingcognitive state may be identified. The similarity between a plurality ofdata points on the captured facial image may be compared between thestored facial image and the captured facial image and on a scale of0-100. For example, 90/100 data points matching would result in acognitive state match score of 90%. If the cognitive state match scoreis at least 90% when the identification threshold is set to 90% then amatch is made of the corresponding cognitive state. In the exampleillustrated in FIG. 7, the threshold is set at 90% and the profile matchscore for cognitive state 1 is 98%, and thus cognitive state 1 isidentified. However, exemplary embodiments of the present invention arenot limited thereto, and cognitive state analysis may be performed asdesired.

According to an exemplary embodiment of the present invention, detectingand characterizing a cognitive state of a user may include analyzinggaze activity, facial expressions, body motions, and/or voice variation,etc. Deep learning, visual analytics, and other statistical algorithmssuch as simple k-nearest neighbor classifier, and/or SVM may be employedin detecting an characterizing cognitive states.

The current cognitive state of the user may indicate that the targetuser requires emergency assistance, and the task may include the aerialdrone companion device 100 contacting a third party. For example a 911operator may be contact through a cell phone communication network bythe aerial drone companion device 100. Alternatively, police or firerescue services may be contacted without the need for the user sayinganything (e.g., a cognitive state of “in-pain” or “distressed” may beidentified). The microphone 102 and the speaker 110 of the aerial dronecommunication device 100 may be used to enable a call between emergencyservices and a user without the need for the user to pick up a phone.The aerial drone communication device 100 may transmit text messages oremails, such as through voice to text communication from the user.

According to an exemplary embodiment of the present invention, inaddition to recognizing cognitive states such as happy or sad based onfacial expressions, the cognitive state identification module 701 mayidentify user postures, such as standing upright, laying in bed, orfallen down and likely injured. For example, an elderly user may beidentified as being on the ground (i.e., not in bed), unresponsive andwith their eyes closed. This may be detected by determining a bodyposition and an inability to detect a gaze of the user. Law enforcementor emergency personnel may be contact by the aerial drone companiondevice 100 (e.g., through a cell tower communication network).

According to an exemplary embodiment of the present invention, a taskmay be modified based on a cognitive state. For example, if a sadcognitive state is detected, the aerial drone communication device 100may tell a joke, play uplifting music or bring the user a particularobject. For example, a column for cognitive state may be added to thetable described below with reference to FIG. 8, and may be used tomodify user preferences and thus to modify a task signal.

FIG. 8 illustrates a method of storing and accessing user preferences byan aerial drone companion device according to an exemplary embodiment ofthe present invention.

Referring to FIG. 8, the interaction information hub 124 and/or 324 maygenerate and store a user interaction and preference table 801. Forexample, the computer (e.g., 120 or 320) may determine that User Aprefers upbeat music in the evening and slow paced music in the morning.

According to an exemplary embodiment of the present invention, the tasksignal generated by the computer (e.g., 120 or 320) may be further basedon the list of personalized preferences of the first user. A list ofexemplary user preferences is illustrated in the user interaction andpreference table 801 described with reference to FIG. 8.

According to an exemplary embodiment of the present invention, thesecond voice command may direct the aerial drone companion device 100 toplay music. The task signal may indicate a type of music to be playedbased on music preferences of the first user stored in the list ofpersonalized preference of the first user and on music preferences ofthe second user stored in the list of personalized preferences of thesecond user.

According to an exemplary embodiment of the present invention, themethod of operating an aerial drone companion device 100 may includecapturing at least one image of the first user by the image capturedevice 103. The method of operating an aerial drone companion device 100may include transmitting the at least one image of the first user fromthe aerial drone companion device 100 to the computer (e.g., 120 or320), which may be located remotely from the aerial drone companiondevice 100. The method of operating an aerial drone companion device 100may include storing the at least one image of the first user in a firstuser record corresponding to the first user in a user database 323. Themethod of operating an aerial drone companion device 100 may includegenerating a list of personalized preferences of the first user usingthe second voice command and previous voice commands spoken by the firstuser. The method of operating an aerial drone companion device 100 mayinclude storing the list of personalized preferences of the first userin the first user record (see, e.g., the user interaction and preferencetable 801 described with reference to FIG. 8). Thus, the task signalgenerated by the computer (e.g., 120 or 320) may be further based on thelist of personalized preferences of the first user.

FIG. 9 illustrates a method of operating an aerial drone companiondevice according to an exemplary embodiment of the present invention.

Referring to FIG. 9, a method of operating an aerial drone companiondevice 100 may include receiving an input 901 at the aerial dronecompanion device 100. The input may include a voice or gesture command.A direct request 902 may be detected by the aerial drone companiondevice 100. If the request is detected a learning technique may beapplied to generate a task for a given request 903. The learningtechnique may include a comparison with past user interactions (see,e.g., the user interaction and preference table 801 described withreference to FIG. 8). If the request is not detected, or is not clearlydetected a request clarification may be sent by the aerial dronecompanion device 100 (e.g., by using the light array 111 or the speaker110). Relevant context and conditions for the task may be determined905. One or more learning techniques may be applied to determine acurrent user context 906. For example, a user group (e.g., elderlyperson or child); a current user context (e.g., cognitive state); and/orsocial context (e.g., outdoors or indoors in a crowded space) may bedetermined. The task may be optimized based on the learning performedand the context determined 907. The task may be executed by the drone908. The aerial drone companion device 100 may orient itself based onthe learning performed or the context determined (e.g., to clarify thetask) 909.

According to an exemplary embodiment of the present invention, if therequest was not clarified (e.g., at block 904) then a task similaritymatching procedure may be applied to determine if the current taskrequest is similar to a past task request 910. If the task can then beidentified, then the procedure will continue from block 905. If the taskis still not identified, then the aerial drone companion device 100 mayfly closer to the user and update the user on the status of therequested task 911. The aerial drone companion device 100 may provideone or more recommendations to the user 912.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may determine an optimal or improvedway for the aerial drone companion device 100 to learn for an individualor for a group in a cohort. The aerial drone companion device 100 mayorient toward the user appropriately (e.g., closer for someone withhearing impairment to hear sounds emitted from the aerial dronecompanion device 100). The aerial drone companion device 100 mayintelligently store or ingest learned information by one or more droneand intelligently respond to new user command (e.g. speech, gesture) andmay compose/orchestrate activities to maximize the owner's productivitybased on user cohort (e.g. busy schedule fetched from the user Calendar)and user patterns. A personalized longitudinal history (e.g., generate adiary about a child for the parent based on the child historicactivities or interactions with the drone) may be generated.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may assess automatically orchestrategroup interest in a home by detecting individual people (e.g., fromtheir voice), learning their common interest and suggesting commonactivity (or based on majority interest). For example, the drone maysay: “It is Sunday Afternoon, you all like the new movie by LeonardoDiCaprio on Netflix”. The aerial drone companion device 100 may initiateinteraction/conversation (via phone or video conference) with person'spersonal doctor based on the analysis of the user cohort. Thiscapability of the drone may be set (e.g., ON or OFF) by the owner of theuser and the aerial drone companion device 100 may communicate (e.g., toget permission via command) with the person prior to initiating saidinteraction or conversation. The aerial drone companion device 100 mayfurther be equipped with virtual reality console powered by WatsonPersonality Analytics and activated by the drone based learning of usercohort.

According to an exemplary embodiment of the present invention, theaerial drone companion device 100 may be shared with a number of people(e.g., a family). Thus, over a time aerial drone companion device 100may change modes based on different interaction pattern per user. Theaerial drone companion device 100 may recognize individual users livingin a single home by applying speech recognition techniques, or gesturebased interaction through visual analytics. The aerial drone companiondevice 100 may maintain and update personalized preferences or styles ofindividual users. In the case of multiple users present at the sametime, the aerial drone companion device 100 may apply similarityanalysis (e.g., clustering) to identify and play/recommend commoninterests (e.g. songs, video) to the family. The aerial drone companiondevice 100 may support and recognize multimodal user interactions, andrespond to it appropriately based on the context. The event logsgenerated through these interactions may be uploaded into theInteraction Information Hub 124 or 324 (e.g., in the Cloud) for furtheranalytics and personalization.

The orientation of the aerial drone companion device 100 with respect tothe speaker or user may be a change in direction of the aerial dronecompanion device 100, a flight towards the speaker or user. The aerialdrone companion device 100 may follow the user when the user is walkingaround the home, or taking a walk on the user's property. The aerialdrone companion device 100 may include 3D spatial and other features. Byremoving the need to use buttons, dials and switches, consumers caneasily operate appliances or give commands, with their hands full orwhile doing other tasks.

The aerial drone companion device 100 may follow the user and optionallyland when the user is still. The aerial drone companion device 100 maypick up small objects (e.g., with the object manipulation arm 104) andtake video (e.g., with the image capture device 103). The aerial dronecompanion device 100 may be capable of voice interaction, music playback(e.g., from a location you ask the drone to fly to), making to-do lists,setting alarms (e.g., from a location you ask the drone to fly to),streaming podcasts, playing audio books, and providing weather, trafficand other real time information.

According to an exemplary embodiment of the present invention thespeaker 110 may be a Bluetooth speaker, which may play music from aconnected device, such as a Smartphone. The aerial drone companiondevice 100 may respond to a user's questions about items in anelectronic calendar. The aerial drone companion device 100 may matchquestions with existing Q&A (e.g., from past interactions) to see ifthey can be answered. If not, the aerial drone companion device 100 mayseek a remote response from the interaction information hub 124 or 324.

According to an exemplary embodiment of the present invention, thevisual systems of the aerial drone companion device 100 may be highlysensitive to motion. The motion processing in the aerial drone companiondevice 100 may be similar to an insects' optomotor response. Thisresponse, which is a turning response evoked by the apparent movement ofthe visual environment, serves to stabilize the aerial drone companiondevice's orientation with respect to the environment and the user, andto be more responsive to the user. The optomotor response may becomplemented by audio processing and source localization via multiplesensors and methods that use the time difference of arrival (TDOA)technique. Interaural time differences (interaural phase differences)and interaural level differences play a role for the hearing of manyanimals.

According to an exemplary embodiment of the present invention, thevisual systems of the aerial drone companion device 100 may have TVintegration. The aerial drone companion device 100 may serve thefunction of a television remote control.

According to an exemplary embodiment of the present invention, thevisual systems of the aerial drone companion device 100 may includethermostat integration. For example, the following interaction may occur“Drone, it's cold.” “Sorry, Cliff, let me turn up the temperature.” Theaerial drone companion device 100 may sync with smartphone apps.

FIG. 10 illustrates an example of a computer system capable ofimplementing the methods according to exemplary embodiments of thepresent invention.

The system and method of the present disclosure may be implemented inthe form of a software application running on a computer system, forexample, a mainframe, personal computer (PC), handheld computer, server,etc. The software application may be stored on a recording media locallyaccessible by the computer system and accessible via a hard wired orwireless connection to a network, for example, a local area network, orthe Internet.

The computer system referred to generally as system 1000 may include,for example, a central processing unit (CPU) 1001, random access memory(RAM) 1004, a printer interface 1010, a display unit 1011, a local areanetwork (LAN) data transmission controller 1005, a LAN interface 1006, anetwork controller 1003, an internal bus 1002, and one or more inputdevices 1009, for example, a keyboard, mouse etc. As shown, the system1000 may be connected to a data storage device, for example, a harddisk, 1008 via a link 1007.

FIG. 11 depicts a cloud computing environment according to an embodimentof the present invention. FIG. 12 depicts abstraction model layersaccording to an embodiment of the present invention.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire. Computer readable program instructions described hereincan be downloaded to respective computing/processing devices from acomputer readable storage medium or to an external computer or externalstorage device via a network, for example, the Internet, a local areanetwork, a wide area network and/or a wireless network. The network maycomprise copper transmission cables, optical transmission fibers,wireless transmission, routers, firewalls, switches, gateway computersand/or edge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions (see, e.g., FIGS. 1-9).

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

It is understood in advance that although this disclosure includes adetailed description on cloud computing, implementation of the teachingsrecited herein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g. networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure comprising anetwork of interconnected nodes.

Referring now to FIG. 11, illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 comprises one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 11 are intended to be illustrative only and that computing nodes10 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 12, a set of functional abstraction layersprovided by cloud computing environment 50 (FIG. 11) is shown. It shouldbe understood in advance that the components, layers, and functionsshown in FIG. 12 are intended to be illustrative only and embodiments ofthe invention are not limited thereto. As depicted, the following layersand corresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networks andnetworking components 66. In some embodiments, software componentsinclude network application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

In one example, management layer 80 may provide the functions describedbelow. Resource provisioning 81 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may comprise applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; and a voice command analysis module 96, whichmay perform, for example. NLP, as discussed above in more detail.

While the present invention has been particularly shown and describedwith reference to exemplary embodiments thereof, it will be understoodby those of ordinary skill in the art that various changes in form anddetail may be made therein without departing from the spirit and scopeof the present invention as defined by the following claims.

1. A method of operating an aerial drone companion device, comprising:detecting a first voice command spoken by a first user by at least onemicrophone disposed on the aerial drone companion device; autonomouslyorientating the aerial drone companion device such that an image capturedevice disposed on the aerial drone companion device faces the firstuser in response to detecting the first voice command; detecting asecond voice command spoken by the first user by the at least onemicrophone while the image capture device faces the first user and whilethe first user looks at the image capture device; transmitting thesecond voice command from the aerial drone companion device to acomputer located remotely from the aerial drone companion device;receiving a task signal indicating a task to be performed, wherein thetask signal is generated by the computer based on the second voicecommand, and the task signal is transmitted by the computer and receivedby the aerial drone companion device; and autonomously executing thetask by the aerial drone companion device.
 2. The method of claim 1,wherein the first voice command is a wakeup command that causes theaerial drone companion device to transition from a low-power operationmode to a normal operation mode.
 3. The method of claim 2, wherein theaerial drone companion device is stationed on ground while in thelow-power operation mode, autonomously flies above the ground andfollows the first user while in the normal operation mode and while theuser is moving, and is stationed on the ground while in the normaloperation mode and while the user is not moving.
 4. The method of claim2, wherein the wakeup command is predefined by the first user.
 5. Themethod of claim 1, wherein autonomously orientating the aerial dronecompanion device comprises at least one of autonomously flying theaerial done companion device toward the first user and autonomouslyrotating the aerial drone companion device such that the image capturedevice faces the first user.
 6. The method of claim 1, wherein thesecond voice command indicates a destination at which the task is to beexecuted, and the aerial drone companion device autonomously flies tothe destination to execute the task in response to receiving the tasksignal.
 7. The method of claim 6, wherein the second voice commandfurther indicates a time at which the task is to be executed, and theaerial drone companion device autonomously flies to the destination atthe indicated time to execute the task in response to receiving the tasksignal.
 8. The method of claim 1, further comprising: generating a voicemodel of the first user using at least one of the first voice command,the second voice command, and additional voice commands spoken by thefirst user; storing the voice model of the first user in a first userrecord corresponding to the first user in a user database; generating alist of personalized preferences of the first user using the secondvoice command and previous voice commands spoken by the first user; andstoring the list of personalized preferences of the first user in thefirst user record.
 9. The method of claim 8, wherein the task signalgenerated by the computer is further based on the list of personalizedpreferences of the first user.
 10. The method of claim 8, furthercomprising: creating a voice model of a second user using at least onevoice command spoken by the second user; storing the voice model of thesecond user in a second user record corresponding to the second user inthe user database; generating a list of personalized preferences of thesecond user using the at least one voice command spoken by the seconduser; storing the list of personalized preferences of the second user inthe second user record; and determining that the first and second userare present, using the at least one microphone, when the second voicecommand spoken by the first user is detected by the at least onemicrophone, wherein the task signal generated by the computer is furtherbased on the list of personalized preferences of the first user and thesecond user.
 11. The method of claim 10, wherein the second voicecommand directs the aerial drone companion device to play music, and thetask signal indicates a type of music to be played based on musicpreferences of the first user stored in the list of personalizedpreference of the first user and on music preferences of the second userstored in the list of personalized preferences of the second user. 12.The method of claim 1, further comprising: capturing at least one imageof the first user by the image capture device; transmitting the at leastone image of the first user from the aerial drone companion device tothe computer located remotely from the aerial drone companion device;storing the at least one image of the first user in a first user recordcorresponding to the first user in a user database; generating a list ofpersonalized preferences of the first user using the second voicecommand and previous voice commands spoken by the first user; andstoring the list of personalized preferences of the first user in thefirst user record.
 13. The method of claim 12, wherein the task signalgenerated by the computer is further based on the list of personalizedpreferences of the first user.
 14. The method of claim 1, furthercomprising: capturing at least one image of the first user by the imagecapture device; and determining a current cognitive state of the firstuser using the at least one image, wherein the task signal generated bythe computer is further based on the current cognitive state of thefirst user.
 15. The method of claim 1, wherein the task comprisescommunicating with an appliance located remotely from the aerial dronecompanion device to execute a function of the appliance.
 16. The methodof claim 1, wherein the task comprises placing an order for goods orservices by relaying the order for goods or services from the aerialdrone companion device to a seller or provider of goods or serviceslocated remotely from the aerial drone companion device.
 17. The methodof claim 1, further comprising: filtering noise generated by a propellerof the aerial drone companion device to identify the first and secondvoice commands spoken by the first user.
 18. A method of operating anaerial drone companion device, comprising: capturing at least oneregistration image of a user by an image capture device disposed on theaerial drone companion device; registering the user as a target user ofthe aerial drone companion device using the at least one registrationimage; autonomously flying the aerial drone companion device from acharging station to the target user, wherein the aerial drone companiondevice locates the target user by capturing images using the imagecapture device while flying, and by comparing people in the capturedimages to the at least one registration image until the target user isidentified, wherein the aerial drone companion device cycles betweenautonomously docking in the charging station and autonomously flying tothe target user at a predetermined interval; determining a currentcognitive state of the target user by capturing at least one image ofthe target user using the image capture device and comparing the atleast one image to a plurality of predefined images in an imagedatabase; and autonomously executing a task based on the cognitive stateof the target user without receiving input from the user.
 19. The methodof claim 18, wherein the current cognitive state indicates that thetarget user requires emergency assistance, and the task comprises theaerial drone companion device contacting a third party.
 20. An aerialdrone companion device, comprising: a processor; at least one microphonethat detects a first voice command and a second voice command spoken bya user under control of the processor; an image capture device; aplurality of propellers that autonomously orientate the aerial dronecompanion device, under control of the processor, such that the imagecapture device faces the user in response to the at least one microphonedetecting the first voice command, wherein the second voice command isdetected by the at least one microphone while the image capture devicefaces the first user and while the first user looks at the image capturedevice; and a network adapter that transmits the second voice commandfrom the aerial drone companion device to a computer located remotelyfrom the aerial drone companion device under control of the processor,and that receives a task signal from the computer indicating a task tobe performed, wherein the task signal is generated by the computer basedon the second voice command, wherein the aerial drone companion deviceautonomously executes the task.